ASoC: SOF: Initial support for IPC4 compress offload#5647
ASoC: SOF: Initial support for IPC4 compress offload#5647ujfalusi wants to merge 23 commits intothesofproject:topic/sof-devfrom
Conversation
|
I don't see these errors with |
cea54e7 to
c3f4c47
Compare
|
Will test this on IMX platforms, but we are only using IPC3 so not sure how much it helps. I'm not working on moving IMX to IPC4 but it will take a while. Will be back with the results asap. |
@dbaluta, we want to make sure that we don't break iMX (IPC3) and we are on reverse grounds that we can only test the new IPC4 support.
Thank you! |
lgirdwood
left a comment
There was a problem hiding this comment.
LGTM, some minor opens, some rebasing/squashing probably needed.
| /* | ||
| * stream_start_offset is updated to memory window by FW based on | ||
| * pipeline statistics and it may be invalid if host query happens before | ||
| * the statistics is complete. And it will not change after the first initiailization. |
There was a problem hiding this comment.
may need a comment on whether header is included or not.
There was a problem hiding this comment.
Header has no meaning in here, this is pure PCM data, exactly like normal PCMs, we count the samples played out.
|
|
||
| out: | ||
| caps->direction = cstream->direction; | ||
| caps->min_fragment_size = 3 * 1024; |
There was a problem hiding this comment.
based on any particular format ? Maybe worth a comment
There was a problem hiding this comment.
gut feeling ;) I will add defines :D
|
@ujfalusi Tested this today with sof-dev branch and I get some problems but could be unrelated to the change. Need to check if we have some internal patches not upstreamed. |
|
@ujfalusi Tested the patches with IMX8MP using IPC3 compress implementation with MP3 and AAC and it works fine. I will add my comments to the code later. |
228cd7c to
a57957e
Compare
|
Changes since v2:
|
52d57d7 to
f992ff0
Compare
|
Changes since v3:
|
f992ff0 to
f16f40d
Compare
|
Changes since v4: It is a surprise that this did not blow up, but it somehow managed to 'work', just that the min_fragment_size turned 0 after the compr_open (as we used the data for read only). Added a helper to calculate on spot the minimal fragment size instead, it is used only during open operation. |
f16f40d to
fadeee4
Compare
|
Changes since v5:
|
11cb387 to
e22d68e
Compare
|
Changes since v6:
There is not yet corresponding firmware PR, the WIP branch for testing is https://github.com/ujfalusi/sof/commits/peter/topic/ipc4_compr |
e22d68e to
bc8f6ba
Compare
|
Changes since v7:
With this version end of file drain works correctly without errors from BE. |
|
Changes since v8:
|
I need to move patches around and squash them, also aligning on the param ids in firmware config, plus the notification magic for EOS completion. |
f21f5f1 to
b7122bb
Compare
|
Changes since v11:
|
There was a problem hiding this comment.
Pull request overview
Adds initial support for ALSA compress-offload on SOF IPC4 (Intel MTL+) by wiring new compress stream ops through the SOF core, ASoC compress/DPCM paths, IPC4 topology/init extensions, and Intel HDA platform glue.
Changes:
- Introduces IPC4 compress offload implementation (open/free/set_params/trigger/pointer/copy) and firmware codec capability discovery.
- Extends SOF/ASoC plumbing to support compress streams in widget setup, triggers (incl. EOS/DRAIN), and BE runtime handling.
- Adds Intel HDA compressed-stream platform ops and updates IPC4 notification handling for drain completion.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| sound/soc/sof/topology.c | Treat decoder/encoder widgets like effects during widget readiness/token validation. |
| sound/soc/sof/sof-priv.h | Add DSP ops callbacks for compressed streams. |
| sound/soc/sof/sof-audio.h | Add process-widget macro; extend pcm_ops signatures; add compress ops pointer and per-stream compress params storage; expose helpers used by compress. |
| sound/soc/sof/sof-audio.c | Harden widget free path; refactor widget-free recursion; move/commonize compress elapsed + page table helpers. |
| sound/soc/sof/pcm.c | Export widget-setup/helpers; adapt pcm_ops hw_free/trigger signatures; select compress_ops from IPC-specific pcm ops. |
| sound/soc/sof/ops.h | Add SOF platform wrapper helpers for compressed stream ops. |
| sound/soc/sof/ipc4.c | Route IPC4 module notifications for ALSA controls vs compress drain completion. |
| sound/soc/sof/ipc4-topology.h | Add copier feature bit definition; extend process struct for init_ext module data. |
| sound/soc/sof/ipc4-topology.c | Enable fast mode for compress copier; add init_ext module_data object support; handle decoder/encoder widgets as process modules. |
| sound/soc/sof/ipc4-priv.h | Store codec info pointer; move timestamp struct to header; declare IPC4 compress ops + drain-done hook. |
| sound/soc/sof/ipc4-pcm.c | Extend pipeline triggering for EOS (drain); add locking/robustness checks; export timestamp helpers; add compress_ops pointer to ipc4_pcm_ops. |
| sound/soc/sof/ipc4-loader.c | Query firmware “SOF info” tuples and cache codec capability data. |
| sound/soc/sof/ipc4-compress.c | New IPC4 compress-offload implementation (caps/params/trigger/pointer/copy + drain notify). |
| sound/soc/sof/ipc3-priv.h | Declare IPC3 compressed ops symbol when compress enabled. |
| sound/soc/sof/ipc3-pcm.c | Update hw_free/trigger signatures; wire IPC3 compress ops into pcm ops table. |
| sound/soc/sof/ipc3-compress.c | Rename/export IPC3 compress ops; switch to shared elapsed/page-table helpers. |
| sound/soc/sof/intel/hda.h | Add prototypes for Intel HDA compressed-stream platform ops. |
| sound/soc/sof/intel/hda-stream.c | Share hw_free helper between PCM and compress; add compressed-stream LLP accessor. |
| sound/soc/sof/intel/hda-pcm.c | Implement HDA compressed open/close/hw_params/trigger/pointer hooks. |
| sound/soc/sof/intel/hda-common-ops.c | Wire HDA compressed ops into the SOF HDA common dsp_ops table. |
| sound/soc/sof/intel/Kconfig | Adjust Intel platform selects, enabling SOF compress where needed (e.g., TGL/MTL). |
| sound/soc/sof/core.c | Initialize platform component driver after IPC ops are ready (non-dspless). |
| sound/soc/sof/Makefile | Build IPC3/IPC4-specific compress objects under CONFIG_SND_SOC_SOF_COMPRESS. |
| sound/soc/soc-pcm.c | Allocate/free BE runtimes for FE compress streams in DPCM startup/stop paths. |
| sound/soc/soc-compress.c | Rework FE/BE trigger ordering and ensure STOP on free when draining. |
| sound/core/compress_offload.c | Track open compress files; stop/wake streams on device disconnect; manage module refs. |
| include/sound/sof/ipc4/header.h | Add FW cfg IDs for SOF info and compress notify magic; extend init-ext object IDs. |
| include/sound/compress_driver.h | Add open_list to snd_compr for tracking open compress files. |
Comments suppressed due to low confidence (1)
sound/soc/soc-pcm.c:1743
- In
dpcm_be_dai_startup(), whenfe->fe_compris true you allocatebe_substream->runtimefor each BE, but if__soc_pcm_open(be, be_substream)fails yougoto unwindwithout freeing that newly allocated runtime. The rollback macro intentionally stops up to (but not including)last, so the failing BE’s runtime will leak andbe_substream->runtimeremains non-NULL. Freebe_substream->runtime(and set it to NULL) on the__soc_pcm_open()error path whenfe->fe_compris set.
if (!fe->fe_compr) {
be_substream->runtime = fe_substream->runtime;
} else {
be_substream->runtime = kzalloc(sizeof(*be_substream->runtime), GFP_KERNEL);
if (!be_substream->runtime) {
err = -ENOMEM;
goto unwind;
}
}
err = __soc_pcm_open(be, be_substream);
if (err < 0) {
be->dpcm[stream].users--;
if (be->dpcm[stream].users < 0)
dev_err(be->dev, "ASoC: no users %s at unwind %s\n",
snd_pcm_direction_name(stream),
dpcm_state_string(be->dpcm[stream].state));
be->dpcm[stream].state = SND_SOC_DPCM_STATE_CLOSE;
goto unwind;
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| struct snd_soc_pcm_runtime *rtd = cstream->private_data; | ||
| struct snd_sof_pcm *spcm; | ||
| /* TODO: we don't query the supported codecs for now, if the | ||
| * application asks for an unsupported codec the set_params() will fail. | ||
| */ | ||
| spcm = snd_sof_find_spcm_dai(component, rtd); | ||
| if (!spcm) | ||
| return 0; |
There was a problem hiding this comment.
sof_ipc4_compr_get_params() returns 0 when snd_sof_find_spcm_dai() fails, which reports success to the compress core while leaving *params untouched/undefined. This should return an error (e.g. -EINVAL/-ENODEV) like the other IPC4 compress callbacks when the stream/PCM can’t be found.
| struct snd_soc_pcm_runtime *rtd = cstream->private_data; | |
| struct snd_sof_pcm *spcm; | |
| /* TODO: we don't query the supported codecs for now, if the | |
| * application asks for an unsupported codec the set_params() will fail. | |
| */ | |
| spcm = snd_sof_find_spcm_dai(component, rtd); | |
| if (!spcm) | |
| return 0; | |
| struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component); | |
| struct snd_soc_pcm_runtime *rtd = cstream->private_data; | |
| struct snd_sof_pcm *spcm; | |
| /* TODO: we don't query the supported codecs for now, if the | |
| * application asks for an unsupported codec the set_params() will fail. | |
| */ | |
| spcm = snd_sof_find_spcm_dai(component, rtd); | |
| if (!spcm) { | |
| dev_err(sdev->dev, "%s: can't find spcm\n", __func__); | |
| return -EINVAL; | |
| } |
| if (crtd->buffer_size < SOF_IPC4_COMPR_MIN_BUFFER_SIZE(min_fragment_size)) { | ||
| dev_err(dev, "%s: Too small buffer size %llu (minimum is %u)", | ||
| __func__, crtd->buffer_size, | ||
| SOF_IPC4_COMPR_MIN_BUFFER_SIZE(min_fragment_size)); |
There was a problem hiding this comment.
dev_err() format string is missing a trailing newline, which leads to log lines being concatenated. Please add \n at the end of the message.
| if (fragments < SOF_IPC4_COMPR_MIN_FRAGMENTS) { | ||
| dev_err(dev, | ||
| "%s: Insufficient amount of fragments: %llu (minimum is %d)", | ||
| __func__, fragments, SOF_IPC4_COMPR_MIN_FRAGMENTS); | ||
| return -EINVAL; |
There was a problem hiding this comment.
dev_err() format string is missing a trailing newline, which leads to log lines being concatenated. Please add \n at the end of the message.
| * To handle multiple pause/resume/xrun we need to add | ||
| * the positions to simulate how the firmware behaves | ||
| * the positions to simulate how the firmware behaves. | ||
| * Chained DAM does not support compress streams. We should |
There was a problem hiding this comment.
Typo in comment: "Chained DAM" should be "Chained DMA".
| * Chained DAM does not support compress streams. We should | |
| * Chained DMA does not support compress streams. We should |
| static inline snd_pcm_uframes_t | ||
| snd_sof_compr_platform_pointer(struct snd_sof_dev *sdev, | ||
| struct snd_compr_stream *cstream, | ||
| struct snd_compr_tstamp64 *tstamp) | ||
| { | ||
| if (sof_ops(sdev) && sof_ops(sdev)->compr_pointer) | ||
| return sof_ops(sdev)->compr_pointer(sdev, cstream, tstamp); | ||
|
|
||
| return 0; | ||
| } |
There was a problem hiding this comment.
snd_sof_compr_platform_pointer() is declared to return snd_pcm_uframes_t, but the underlying DSP op compr_pointer (and callers like sof_ipc4_compr_pointer()) expect an int error code. This will break negative error propagation (e.g. -EINVAL becomes a large positive) and can cause callers to treat failures as success. Change the wrapper return type to int (and keep returning 0 when unimplemented).
| int hda_dsp_compr_pointer(struct snd_sof_dev *sdev, struct snd_compr_stream *cstream, | ||
| struct snd_compr_tstamp64 *tstamp) | ||
| { | ||
| struct hdac_stream *hstream = cstream->runtime->private_data; | ||
|
|
||
| /* hstream->curr_pos is updated when we receive the ioc */ | ||
| tstamp->copied_total += hstream->curr_pos; | ||
|
|
||
| tstamp->byte_offset = hda_dsp_stream_get_position(hstream, cstream->direction, true); | ||
|
|
||
| return 0; |
There was a problem hiding this comment.
hda_dsp_compr_pointer() adds hstream->curr_pos to tstamp->copied_total on every call. Since hstream->curr_pos is already cumulative (it is incremented in hda_dsp_compr_bytes_transferred()), using += will double-count bytes as userspace polls the pointer. Assign tstamp->copied_total from hstream->curr_pos instead of accumulating into the passed-in value.
| while (offset < sof_info_size) { | ||
| tuple = (struct sof_ipc4_tuple *)((u8 *)sof_info_data + offset); | ||
|
|
||
| switch (tuple->type) { | ||
| case SOF_IPC4_SOF_CODEC_INFO: | ||
| ipc4_data->codec_info = devm_kmemdup(sdev->dev, tuple->value, | ||
| tuple->size, GFP_KERNEL); | ||
| if (!ipc4_data->codec_info) { | ||
| ret = -ENOMEM; | ||
| goto out; | ||
| } | ||
| break; | ||
| default: | ||
| break; | ||
| } | ||
|
|
||
| offset += sizeof(*tuple) + tuple->size; | ||
| } |
There was a problem hiding this comment.
sof_ipc4_query_sof_info() walks a tuple array using offset += sizeof(*tuple) + tuple->size without validating that (1) sof_info_size - offset is at least sizeof(*tuple) and (2) tuple->size keeps the next offset within sof_info_size (and doesn’t overflow). A malformed/buggy firmware response could cause out-of-bounds reads or an infinite loop. Add strict bounds checks before dereferencing tuple/tuple->value, and reject tuples with sizes that would exceed sof_info_size.
Take a module reference in snd_compr_open() and release it in snd_compr_free(). This pins the card driver module for the lifetime of an open compress stream and prevents card removal while the stream file is still in use. Adjust the open() cleanup paths to drop the added module reference only when it was acquired, and keep release ordering safe by dropping the module reference before freeing stream data. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Track open compressed streams per device so disconnect can stop active streams and wake waiters before snd_unregister_device(). This aligns compressed stream teardown with PCM disconnect behavior and prevents active userspace streams from running into unregister races. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
With DPCM when compr is used on FE side, the BE is still running as 'normal' PCM. It is expected that the be_substream->runtime on the BE is not NULL, for example some codec drivers expect to have the substream->runtime valid, to store configuration for example. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
… PCMs The FE-BE trigger sequence should be dynamic, similarly how soc-pcm.c dpcm_fe_dai_do_trigger() does it. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
If the last trigger that the compr device received is a DRAIN then the DPCM is left in running state (no stop trigger is sent). Before we execute the free we need to send a STOP trigger to make sure that both BE and FE is in expected state and prepared for closing. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
In preparation for adding support for compressed offload support for IPC4, rename the current compress implementation with the IPC3 prefix. Introduce a new field in struct sof_ipc_pcm_ops to save the IPC-specific compressed ops pointer. This should be set when the component driver ops are assigned during SOF device probe. Expose a couple of common functions that will be used by both IPC-specific implementations and rename the compress.c file to ipc3-compress.c Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Serialize trigger/free with pipeline_state_mutex and validate pipeline entries before use. Also clear pipeline_list->count when freeing lists to avoid stale entries during concurrent teardown. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
During widget FREE traversal, SOF walks DAPM sink paths recursively while widgets and paths can be torn down. This can lead to stale pointer usage in sof_free_widgets_in_path() when path entries disappear during recursion. Harden the FREE path by: - validating scheduler/pipeline pointers before recursive free - using a safe DAPM path iterator for sink traversal - carrying a stable widget-list snapshot through recursion - skipping NULL sink edges during traversal The safe traversal can leave path->walking set on surviving edges after FREE, which may short-circuit later DAPM walks and break consecutive playback open/stop cycles. Reset walking flags for widgets in the current DAPM list after FREE walks (including error paths) to keep subsequent traversals clean. This keeps teardown robust for module-remove race scenarios while preserving normal consecutive playback behavior. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
These are common functions that will also be needed for the IPC4 compressed support. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
In order to reuse the pipeline triggering logic for compressed support with IPC4, modify the signature of the trigger and hw_free PCM IPC ops so that they can be reused. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
…ng stream After the host DMA IID is released, reset the curr_pos to 0 for a clean start for subsequent stream starts. This is not needed for PCM streams but for compressed streams. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Add new ops in the struct snd_sof_ops for platform-specific ops for compresssed streams. Also, define and set them for the HDA platforms. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
The SOF_IPC4_MOD_INIT_DATA_ID_MODULE_DATA type within the module_init_ext area is module specific init data. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
…dules Add support for handling init_ext_module_data for process modules, which is going to be used by decoder and encoder type of process mdoules. The support is generic and it can be extended to other type of process modules or other module types than process with a small update of sof_ipc4_add_init_ext_module_data() function. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
…cm.c The support for compressed stream will also need to have access to the same information which is used for delay reporting for DAI data progression tracking. Make the necessary struct and functions to be available and premare them to be called without a valid substream. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
…_params SOF_INFO (id == 35) tuple holds tuple structured information about SOF features. The first entry in SOF_INFO is the SOF_CODEC_INFO (id == 0) which contains information about the supported codecs for decode/encode in the booted firmware. If present in the fw_config payload, make a copy of it and store it sof_ipc4_fw_data->codec_info to be used by the compressed code. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
The DRAIN trigger is received by a compr device when user space wrote the all data to the buffer and it is waiting for the decoding to be completed. Set the pipeline state to EOS in firmware so it can expect the stream to be stopping anytime soon. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
FAST_MODE allows the host DMA to work in opportunistic, free running mode, which matches with the bitstream nature of compressed devices. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Set and define the compressed ops for IPC4. The initial implementation supports basic features: PAUSE PUSH/RELEASE, DRAIN and progress reporting. Tested with PCM, MP3, AAC and VORBIS codec. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
The decoder module sends a drain done notification when the last chunk of the stream after the EOS from host has been decoded. The notification is a module notification with 0xc0c0 as magic number in event_id upper 16 bit. Call sof_ipc4_compr_drain_done() when the notification arrives to handle it. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
There is no need to select IPC3 and IPC4 along with INTEL_CNL as INTEL_CNL selects both. Similarly, INTEL_MTL selects IPC4, so there is no need to do that for LNL, PTL and NVL. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Select SOF_COMPRESS for TGL and newer platforms, on Intel devices the compressed support is available with IPC4 only. Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Decoder and encoder modules fall under process modules in SOF. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Co-developed-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
4242694 to
787627d
Compare
|
changes since v12:
|
Hi,
Support for compress offload with IPC4 on Intel platforms (MTL+).
This initial version supports basic features, like PAUSE PUSH/RELEASE, DRAIN, progress reporting.
Tested with PCM, MP3, AAC and VORBIS codecs so far with firmware built from the following repo (will be making it's way to sof:main): https://github.com/ujfalusi/sof/commits/peter/topic/ipc4_compr
@dbaluta, if you could be able to test that, we would really appreciate it.
firmware PR (without topology changes): thesofproject/sof#10492 (merged), thesofproject/sof#10534 (open)